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Abstract 

School teams regularly meet to review academic and social problems of individual students, 
groups of students, or their school in general. While the need for problem solving and 
recommendations for how to do it are widely documented, there is very limited evidence 
reflecting the extent to which teams effectively engage in a systematic or effective process at 
these meetings or the extent to which engaging in recommended processes results in positive 
outcomes. One reason there has not been more progress on understanding team-based problem 
solving is the absence of research tools for studying the process and its outcomes. In previous 
research, we developed the Decision Observation, Recording, and Analysis (DORA), 
documented its technical adequacy for assessing the behavior of teams during their team 
meetings, and demonstrated its usefulness in both single-case and randomized controlled trial 
studies examining problem solving in schools. In this research, we expanded DORA to provide 
documentation that the solutions that teams develop have been implemented with fidelity and are 
associated with problem improvement. We found that the revised instrument was a valid and 
reliable tool for assessing team behavior and that it provided technically adequate information on 
the extent to which teams were assessing if they had implemented a solution and if the solution 
made a difference for students. We believe the revised DORA is a measure worthy of use in 
studying team-based problem solving, and we discuss our findings as a base for a well-reasoned 
research agenda for moving the process forward as evidence-based practice. 
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DORA-II Technical Adequacy Brief: 

Measuring the Process and Outcomes of Team Problem Solving 
The use of school teams to identify and solve educational problems is a widely 
recommended and accepted practice in America’s schools (Blankenship, Houston, & Cole, 2010; 
Boudett, City, & Murnane, 2006a,b; Boudett & Steele, 2007; Chenoweth, 2010; Coburn & 
Turner, 2012; Deno, 2005; Leithwood, 2010; Murnane, Boudett, & City, 2009; Steele & Boudett, 
2008a,b). Driven in large measure by a conceptual framework for data- informed decision making 
embodied in No Child Left Behind ( NCLB ) and by reauthorizations of Public Law 94-142 now 
codified as the Individuals with Disabilities Education Improvement Act of 2004, this practice is 
a core feature of both general and special education (Brown-Chidsey & Steege, 2005; Boudett et 
al., 2006a,b; Henley, Ramsey, & Algozzine, 2009; McLeskey, Waldron, Spooner, & Algozzine, 
2014; U. S. Department of Education, 2009). In fact, nearly every school in the country has 
teams meeting regularly to solve administrative as well as academic and social problems. The 
assumption is that problem solving by a group of individuals (including family members and 
students) acting as a team will provide better outcomes than these same individuals acting alone; 
and, an impressive literature exists with guidance and recommendations about the need for and 
the process of team problem solving (Bransford & Stein, 1993; Carroll & Johnson, 1990; 
D’Zurilla, Nezu, & Maydeu-Olivares, 2004; Gilbert, 1978; Huber, 1980; Jorgensen, Scheier, & 
Fautsko, 1981; Marsh, McCombs, & Martorell, 2010; Newton, Horner, Algozzine, Todd, & 
Algozzine, 2009; Tropman, 1996; Tropman & Mills, 1980; Ysseldyke et al., 2006). Important 
takeaways in what has been written about this practice are that (a) the cycle of steps or stages for 
effective problem solving remain impressively consistent across time, context, and authors; (b) 
simply giving teams steps for the process does not guarantee they will effectively use them; (c) 
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current problem solving practices in schools leave much room for improvement; (d) barriers to 
effective problem solving are the same as those often cited as challenges to other promising 
practices (i.e., attitudes and a lack of time, resources, and training); and, (e) there is an 
impressive lack of evidence-based research on the extent to which school teams engage in 
recommended problem solving practices and if those practices are related to positive student 
outcomes. We believe one reason for this paucity of empirical evidence is the absence of well- 
documented data collection tools needed to assess problem solving practices by school teams. 
Importance of Data in Problem Solving 

Data are the currency of team problem solving and associated efforts to reform 
education; in fact, “using data to inform instruction,” “data-based decision making,” “data- 
driven decision making,” “data-infonned decision making,” and “data use” are increasingly 
popular mantras in which policy makers and other professionals place faith in their efforts to 
transform practice (Coburn & Turner, 2012; Cook & Brown, 1999; Gilbert, Compton, Fuchs, & 
Fuchs, 2012; Goren, 2012; Halverson, Grigg, Prichett, & Thomas, 2005; Hamilton et al., 2007; 
Honig & Cobum, 2008; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Means, Padilla, & 
Gallagher, 2010; Moss, 2012; Shinn, 1989; Spillane, 2012; Supovitz, Foley, & Mishook, 2012; 
U. S. Department of Education, 2009; Wayman, Cho, Jimerson, & Spikes, 2012; Wickens, Lee, 
Liu, & Becker, 2004; Young & Kim, 2010). Despite the centrality of data to team problem 
solving, most professionals agree and recent writing suggests that simply having data is a 
necessary but not sufficient condition for using the information effectively, solving problems, or 
producing change (Choppin, 2002; Feldman & Tung, 2001; Hoover, 2009; Marsh et al., 2010; 
Marsh, Pane, & Hamilton, 2006; Mason, 2002; Scott, Liaupsin, Nelson, & McIntyre, 2005; 
Scott, McIntyre, Liaupsin, Nelson, Conroy, & Payne, 2005; Supovitz & Klein, 2003; U. S. 
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Department of Education, 2009; Wayman et al., 2012): “[D]ata do not objectively guide 
decisions on their own — people do, and to do so they select particular pieces of data to negotiate 
arguments about the nature of problems as well as potential solutions” (Spillane, 2012, p. 114). 

There is growing evidence that administrators, teachers, and other professionals working 
in schools need help with selecting relevant data (Blankenship et al., 2010; Depka, 2010; 
Halverson et al., 2005; Scott, McIntyre, Liaupsin, Nelson, & Conroy, 2004; Spillane, 2012; U. 

S. Department of Education, 2009) and asking the right questions so that problem solving fits 
the local context (Jerald, 2005; Stringfield & Yakimowski-Srebnink, 2005; Wallace Foundation, 
2013; Wickens et al., 2004): “Educational institutions are awash with data, and much of it is not 
used well” (Earl & Katz, 2010, p. 10). They also need training organized around an authentic 
task in which teams learn how to use data and available research to craft, implement, and 
evaluate evidence-based solutions for the problems that they identify (Coburn & Turner, 2012; 
Foegen, Jiban, & Deno, 2007; Gallimore, Ermeling, Saunders, & Goldenberg, 2009; Hill, 2010; 
Honig & Venkateswaran, 2012; Jerald, 2005; Little, 2012; Shah & Freedman, 2011). Otherwise 
district, school, and instructional personnel get very good at using data to talk about and 
document problems; and, often spend more time “admiring problems” than using what they 
know to solve problems (Jerald, 2005). While we support the use of data in efforts to improve 
what goes on in America’s schools and classrooms, we also believe that realizing the powerful 
promises in this work requires documentation and enhancement of what is known about the 
process and outcomes of team problem solving. 

Guiding Principles 

An analysis and review of extant writing revealed that in a variety of formats, team 
problem solving has been a recommended practice for many years (Bergan & Kratochwill, 1990; 
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Boudett et al., 2006b; Deno, 1985, 2005; Hamilton et al., 2009; Spillane, 2012; Tilly, 2008; U. S. 
Department of Education, 2009). Yet, as Little (2012) points out, “little of this writing affords a 
window into the actual practices [professionals] employ as they collectively examine and 
interpret . . . data or the ways in which the contexts of data use come to occupy a central or 
peripheral part of . . . ongoing work life” (p. 144). Put another way, what teams of professionals 
actually do under the broad banner of “problem solving” remains relatively underdeveloped, 
understudied, and unknown. 

Our interest in the process of using data to articulate the nature of problems and produce 
desired change related to them (i.e., problem solving) was grounded in six guiding principles. 

1. The problem-solving process is similar whether engaged by individuals or teams of 

professionals. At the simplest level, people engaged in problem solving collect and 
analyze information, identify strengths and problems, and make changes based on 
what the information tells them. While this is “not rocket science,” doing it well often 
involves the systematic focus and perspective, precision, and persistence of an 
engineer. 

2. The problem-solving process is universally applicable and unbounded by conventions or 

traditions of general and special education. The need to reform practice and improve 
outcomes is not unique to either general or special education. 

3. The problem-solving process is iterative, involving cycles of activities that are 

observable. The process typically involves iterative steps (e.g., collecting information 
from a variety of sources, transforming the information into testable hypotheses, 
selecting, implementing and evaluating interventions that test the hypotheses, and 
adapting or revising the interventions based on fidelity and impact data). 
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4. The problem-solving process is a practice of coordinated activities informed by the 

context in which it occurs and the actions represented there. The process is about 
change and we can only change what we can see. 

5. The problem-solving process is inherently goal-oriented and incomplete until an action 

taken has resulted in a change in behavior. Almost anybody can identify a problem 
and many policy-makers and professionals are remarkably good at doing it; but, the 
work is complete with turning something wrong into something right. 

6. The problem-solving process is defined by actions that are teachable. Giving people data 

does not ensure that they will actively use them to solve problems; the actions of 
problem solving can be and have to be carefully taught. 

We were interested in documenting the quality of problem-solving processes and outcomes 
evidenced in the decision making of teams providing school- wide positive behavioral 
interventions and supports (SWPBIS) in elementary schools. We refer to the process as Team- 
Initiated Problem Solving (TIPS) and operationalized it with a set of actions taken during 
meetings of school-based teams engaged in identifying and resolving students’ social and 
academic behavior problems (see Figure 1). Specifically, team members use TIPS to: 

■ Identify and define students’ social and academic problems with precision (the what, 
when, where, who, and why of a problem); 

■ Establish an objectively-defined goal that, once achieved and maintained, signals 
resolution of the problem to the team’s satisfaction; 

■ Discuss solutions and create a contextually-appropriate implementation plan to resolve 


the problem; 
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■ Implement solutions with a plan for describing, monitoring, and achieving “treatment 
fidelity” of the actions taken; 

■ Monitor the problem across successive meetings to determine the extent to which 
implementation of solution actions with integrity resolves the problem; and, 

■ Revise solution actions as necessary to achieve resolution of the problem. 

A key feature of TIPS is its emphasis on team members’ ongoing and cyclical use of data to 
inform decision making about each stage of the problem-solving process. TIPS also consists of a 
set of “foundational” or “structural” elements that give team members a template for holding 
team meetings that are efficient and effective (e.g., job descriptions for important team roles, a 
standardized meeting minutes form). 

In previous research, we developed the Decision Observation, Recording, and Analysis 
(DORA) tool to document activities and adult behaviors during positive behavior support team 
meetings and to provide a basis for documenting the relationship between teaching these teams 
how to systematically solve problems and achieve improvements in the quality of their school- 
based meetings (Newton, Homer, Todd, Algozzine, & Algozzine, 2012; Newton, Todd, Homer, 
Algozzine, & Algozzine, 2009; Todd, Newton, Horner, Algozzine, & Algozzine, 2009). We 
developed the Decision Observation, Recording, and Analysis-II (DORA-II) to refine our 
understanding of the problem-solving process and to expand our analyses to the outcomes of 
these meetings. In this research, we addressed three objectives: 

1. Replicate key features of the technical adequacy of DORA by documenting the validity 
and reliability of DORA-II for evaluating team problem solving processes. 
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2. Document the technical adequacy of DORA-II for evaluating the extent to which a school 

team assesses the fidelity with which they have implemented an approved solution 
and the impact of its efforts on student outcomes. 

3. Discuss the emerging research agenda that opens up given the existence of a measure that 

allows assessment of team-based problem solving implementation integrity and 
impact. 

We believe using DORA-II provides support for TIPS as an evidence-based practice for teams 
with broad implications for improving problem-solving practices that “permeate all aspects of 
service delivery” in general, special, and remedial education (NASP, 2010, p. 4). 

Method 

Our research took place in two states. We followed widely-used procedures for 
documenting the technical adequacy of measurement data (i.e., scores) obtained using 
assessment instruments in counseling, education, psychology, and other social science areas (cf. 
American Educational Research Association, American Psychological Association, & National 
Council on Measurement in Education, 1999; Dimitrov, 2012; Messick, 1989, 1995; Soukakou, 
2012 ). 

Context 

Over a six-month period, we observed 18 meetings in which team members discussed a 
total of 44 problems in 10 schools that met the following criteria: (a) enrolled elementary- aged 
students, (b) implemented school-wide positive behavioral interventions and supports (SWPBIS; 
Homer, Sugai, & Anderson, 2010) for at least one year, (c) used the School-wide Information 
System (SWIS: May et al., 2010) during the previous six-month period, (d) monitored SWPBIS 
implementation fidelity using procedures recommended by Algozzine et al., 2010) and (e) had 
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SWPBIS team that met at least monthly. Additionally, based on previous experience and 
information provided by external coaches, we selected schools with teams with a wide range of 
experience with the problem-solving process. 

Demographic data for participating schools and PBIS Teams are in Table 1. No 
statistically significant differences (/? > .01) were found between the two states with regard to 
school enrollment, classroom teachers (FTE), student/teacher ratio, percent student enrollment 
for kindergarten through fifth grade, free- or reduced-price lunch, student ethnicity, or student 
gender. Likewise, there was no statistically significant difference ( t= 1.95, df= 7,p > .05) 
between the two states regarding the number of team members; and, teams in both states had 
been implementing PBIS for at least three years. 

Procedure 

We collected data using trained observers who attended team meetings at each of the 
participating schools. Prior to attending these meetings, the observers obtained a copy of the 
minutes from the previous team meeting. The observers’ review of the contents of these minutes 
informed them of (a) any student problems identified by the team and targeted for resolution, and 
(b) any specific solution actions selected by the team to resolve each problem. The observers 
recorded this information on the DORA-II instrument before attending the meeting. During the 
course of the meeting, the observers monitored team members’ discussion of the problems and 
the selected solution actions, and independently recorded data on the DORA-II regarding each 
problem’s solution actions (e.g., whether anyone inquired about implementation of a specific 
solution action, whether anyone reported on implementation of the solution action, team 
members’ description of the implementation status of the solution action, etc.). For cases in 
which no team member reported on a selected solution action, the DORA-II data-collection 
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protocol directed the observer to record “NA/Don’t know” as the implementation status of the 
solution action and no points were awarded. 

Instrumentation. Based on a review of documents addressing the conceptual and 
practical guidance for effective team meetings (see Anderson, 1994; Bradford, 1976; Lencioni, 
2004; Mackin, 2007; Perkins, 2009; Tobia & Becker, 1990) and team-based problem solving, 
we included two sections in the DORA-II. Critical features of the meeting foundations for 
effective problem solving that should be in place at the start, during, and at the end of meetings 
comprise the first part; and the six processes of effective problem solving are represented in the 
second part of the instrument (see Appendix A). 

The “structure” of meetings (e.g., how a team prepares, conducts, and manages the 
follow-up activities) is important to their effectiveness. Critical ‘ foundations ” to be observed at 
the start of a meeting include whether an agenda was distributed, team roles were established, 
team members were present, relevant data were reviewed, and the meeting started “on time.” 
During the meeting, quantitative data should be distributed or projected, the status of one or 
more previous decisions/tasks regarding student social or academic behavior should be reviewed, 
and the fidelity and impact of one or more implemented decisions/tasks regarding student social 
or academic behavior should be discussed. At the close of the meeting, the minutes should be 
distributed; the date and time of the next meeting should be confirmed; and, attendance at the 
beginning and end of the meeting as well as whether it ended “on time” should be recorded. 

Because the process of effective problem solving is iterative, we reasoned that observers 
using DORA-II would also record the cycles of problem-solving and decision-making processes 
used by team members as they address social or academic problems. Each “problem” was 
recorded in a single row that included information about the problem being addressed by the 
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team (e.g., who, what, when, where) and reasons or hypotheses for why it occurring, the type of 
data reviewed, the purpose of the data review, whether the team generated possible solutions for 
solving the problem, the type of action(s) the team decided to implement, the specific action(s) 
the team decided to implement, and the type of evaluation accountability the team documented 
once a decision was reached and recorded. 

DORA-II data were collected in real time by an observer who was present for a full team 
meeting (or at least 70 min). The intent in using and scoring the instrument was to document 
levels of critical features of effective problem solving rather than to record achievement of pre- 
determined standardized or benchmarked scores. 

Design and Data Analysis 

We focused DORA-II on the observable behaviors of team members as they managed 
meetings and identified problems, identified solutions to those problems, and implemented 
and evaluated those solutions. We used multiple methods to evaluate the extent to which the 
use of DORA-II produced data reflective of what we intended it to measure (i.e., validity) and 
the extent to which repeated use of DORA-II produced similar results (i.e., reliability). 

Validity is the extent to which a set of data represents what it purports to represent 
(Suen & Ary, 1989; Thorndike & Thomdike-Christ, 2009). For DORA-II, data are meant to 
reflect the extent to which team members followed the six- step TIPS process including the 
extent to which the process resulted in improvements in identified student academic or social 
behavior problems. Since we were interested in measuring observable behavior representative 
of team decision making, we evaluated the content-related validity of DORA-II to provide 
evidence that information collected was consistent with the underlying knowledge base (i.e., 

The scale contains items that accurately and adequately represent the content of interest). This 
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aspect of the study replicated a core technical adequacy feature of DORA (Objective 1). We 
also extended our previous validity analyses by documenting the extent to which teams 
assessed the fidelity of solutions that were implemented and by confirming the accuracy of 
their reported and actual problem status indicators (Objective 2). 

Reliability is the extent to which a set of data consistently represents what it purports to 
represent (Gall, Borg, & Gall, 1996; Hawkins & Dotson, 1975; Suen & Ary, 1989; Thorndike & 
Thorndike-Christ, 2009). For DORA II, this meant documenting the extent to which scores were 
similar when collected during the same observation by different observers (Objective 1). We 
trained observers using an “observe, review, and revise” cycle. We used sets of scores recorded 
by these trained observers to document overall agreement scores as well as item-by-item 
occurrence agreement indices. We reasoned that reliability was established when occurrence- 
only inter-observer agreement between the two observers on implementation of solution actions 
for associated problems was 85% or better on the foundations and each of the six features. 

Results 

We used multiple methods to address our objectives and to evaluate the extent to 
which the use of DORA-II produced data reflective of what we intended it to measure (i.e., 
validity) and the extent to which repeated use of DORA-II produced similar results (i.e., 
reliability). 

Technical Adequacy Replication 

We evaluated the content-related validity of the DORA-II using a variation of the 
“Content Validity Ratio” (CVR) approach recommended by Lawshe (1975). Our goal was to 
determine the extent of agreement between expected and actual content in our instrument (e.g., 
Are included items addressing areas that are recommended as critical and essential to the 
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effective problem solving?). We cross-tabulated TIPS foundations and processes with data 
collected with DORA-II. We assumed adequate content validity if our content represented 80% 
or more of that included in TIPS-II training. As illustrated in Table 2, DORA-II Foundations 
reflected 87-100% of the core features recommended by experts for effective meetings. We also 
documented complete (100%) agreement between areas of problem solving included on DORA- 
II and the critical features of five widely-accepted problem-solving models (see Table 3). 

We used DORA-II to document critical problem-solving features of meetings where 
team members discussed 12 (27%) new and 32 (73%) old behavior problems. We 
documented the following discrete features in the problems discussed at the meetings: Who 
(84.09%), where (93.18%), what (90.91%), when (84.09%), and why (77.27%). The teams 
focused on group (72.73%) more than individual problems; and, the discussion resulted in a 
“goal for change” ( 1 1 .90%) or an indication of when the problem was expected to change 
(11.36%). The teams followed the TIPS process, and DORA-II was effective at documenting 
the elements of the process. 

As with DORA, we documented inter-rater reliability for DORA-II data with a series of 
analyses. We calculated the percentage of agreement between pairs of observers by comparing 
meeting foundation element scores and decision-making thoroughness scores at 20 randomly- 
selected meetings. Reliability for meeting foundation scores averaged 97% (range 80% to 
100%). We also documented agreement for problem identification, problem precision, 
quantitative data use, goal identification, solution implementation, problem status, and decision 
after status of problem was reported indicators (see Table 4). The average agreement across 
observers for these categories ranged from 74% for solution implementation status to 100% for 
quantitative data use, including agreement of 86% and 83% for problem identification and 
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problem precision, 97% for goal identification, 88% for solution implementation integrity plan, 
89% for status of problem reported, 98% for status of problem compared against goal, and 89% 
for decision after status of problem was reported. Average inter-observer agreement all 
indicators was 90%. 

Technical Adequacy Extension 

DORA-II includes codes for documenting discussions of team members related to 
implementation (i.e., not started, partial implementation, implementation with integrity) and 
impact (i.e., NA New Problem, Worse, No Change, Improved but not to Goal, Improved and 
Met Goal, Unclear, or Not Reported) of solutions implemented to address team-identified 
problems. Participating schools provided evidence of using SWIS for at least six months. This 
created an opportunity for a criterion-related validity analysis (Objective 2) for problems that 
met the following criteria: 

1. Was an Old problem (i.e., a problem with a solution selected by the PBIS Team at a 
meeting previous to the meeting at which the DORA observation was conducted), and 

2. Was an Old problem for which the Primary Observer’s DORA noted that the status of 
the problem was reported by the team (necessary for validity analysis of the Impact 
Score), or 

3. Was an Old Problem for which the Primary Observer’s DORA included a Solution 
Score of either “Partial Implementation” or “Implementation with Integrity” 
(necessary for validity analysis of the Solution Score). 

Sixteen problems discussed by teams met criteria for this part of our study and we 
documented information about partial implementation for 7 (44%) and implementation with 


integrity for 9 (56%) of them. We were interested in the extent to which what team members 
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reported about problem solutions and impact was confirmed by objective data. This additional 
analysis extended our focus from if DORA-II accurately measured what teams decided (e.g., 
about level of solution implementation, and impact of solution implementation on student 
outcomes) to if team assessments could be validated by external documents. For example, 
one school proposed to teach behavioral expectations on the bus, and their confirmation was 
(a) hard copy of the teaching plan, and (b) documented schedule and student performance 
outcomes from the training. Another school team proposed to deliver tokens for appropriate 
behavior in the classroom, but a count of tokens delivered did not indicate they had been 
successful. In both cases an external review of permanent products allowed the researcher to 
confirm the decision of the team that they had “implemented”, “partially implemented” or 
“not implemented” the proposed solution. Of the 16 problems that could be externally 
validated, the results from 13 (81%) of team assessments of implementation confirmed the 
accuracy of the team assessment. 

Similarly, we identified 20 problems where the team was able to make an assessment 
about the impact of the solution on student behavior, and there were external data (SWIS, or 
permanent products) where the observer could confirm if the team assessment of impact was 
accurate (e.g. goal met, progress in desired direction, non-effect). On 18 of the 20 (90%) 
instances the assessment of the team was consistent with the permanent product information. 
One non-agreement was due to the team accurately assessing student progress, but not being 
satisfied that the progress was sufficient, even though it met the initially defined goal. 

Discussion 

Our goal was to document the utility of DORA-II as a measure that may be helpful to 
researchers focused on improving data-based decision making in schools. The extant knowledge 
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base is heavy on making promises but light on documenting processes or products. For example, 
there is plenty of advice on what team members should do as problem-solving practices, but little 
empirical documentation of the extent to which they do them or of their effects on academic or 
social problems that are the reasons for doing them in the first place (Bergan & Kratochwill, 
1990; Blankenship et al., 2010; Boudett et al., 2006a,b; Brown-Chidsey & Steege, 2005; Deno, 
1985, 2005; Earl & Katz, 2010; Leithwood, 2010; Little, 2012; Spillane, 2012; Tilly, 2008). 

In previous research, we documented the technical adequacy of a tool for recording and 
analyzing activities and adult behaviors during positive behavior support team meetings 
(Algozzine, Newton, Homer, Todd., & Algozzine, 2012; Newton, Todd et al., 2009) and used it 
to demonstrate changes that resulted from teaching team members to systematically solve 
problems during their school-based meetings (Newton, Algozzine, Algozzine, Horner, & Todd, 
2011; Newton, Todd et al., 2009; Todd et al., in press; Todd, Horner, Newton, Algozzine, 
Algozzine, & Frank, 2011; Todd, Newton et al., 2009). We developed the DORA-II to confirm 
our understanding of the problem-solving process and to place more emphasis on evaluation of 
implementation fidelity, “solutions,” and “impact.” The findings from our preliminary technical 
analyses reflect that the content of DORA-II is consistent with key components of problem 
solving recommended in the field and provides a basis for measuring each of these features; and, 
our confirmation of team-scored solutions and impact with permanent product extends the value 
of the measure for documenting the ultimate goal of team-based problem solving (i.e., improving 
academic and social behavior). 

Conclusion 

Problem solving is cyclical and goal-oriented. Any protocol for documenting it must 
provide evidence of the extent to which participants engage critical components and stages of 
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the process as well as the extent to which doing so improves problems and benefits students. 
Using DORA-II, we verified both the level of engagement of teams in expected activities and 
the outcomes of those activities. We also confirmed selected core features and processes using 
permanent products obtained and reviewed after the meetings. 

Underlying our work was an interest in determining the extent to which teams engaged 
in core features of effective problem solving. We believe DORA-II has strong potential for 
better understanding problem solving as an evidence-based practice. To date, models for 
problem solving are plentiful but data documenting either use of these models or the impact of 
model use are scarce. Moving problem solving from promise to preferred practice requires 
proof of implementation integrity and impact; that is, most of what we know about problem 
solving is grounded in an opinion-base and stakeholders, policy makers, and other 
professionals need an evidence-base to make informed decisions about education 
interventions. Our findings provide a base for establishing problem solving as an evidence- 
based practice. 

An important limitation to our present analysis is that of the 40 problems assessed 
across the 10 teams, only 16 could be externally confirmed for implementation fidelity and 
only 20 could be externally confirmed for implementation impact. While the results from 
these problems are encouraging (e.g., team-based assessment was confirmed by the external 
assessment), it is possible that teams are more likely to be accurate about implementation and 
impact in situations where clear, external information is available. Further research examining 
team accuracy in their assessment of solution implementation and impact is needed. 

We believe DORA-II to be a valid and reliable index of team problem solving 
processes. By directly observing team behavior with DORA-II researchers are now able to 
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examine questions such as (a) which steps in the problem solving process are essential? (b) 
what data are most helpful to teams in their problem solving?, (c) what features of “solutions” 
improve the likelihood of implementation? and (d) what problem solving practices improve 
the efficiency of team problem solving? The importance of understanding how to guide, teach 
and improve team problem solving is reflected in the large about of time, and resources 
currently spent by educators across the United States. Given the emerging reach and 
sophistication of data access, now more than ever we need research that will help the field 
better understand how to do problem solving with efficiency and impact. DORA-II is one 
measure that may assist researchers to achieve this goal. 
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Table 1 


Demographic Characteristics of Participating Schools and Teams 


Characteristic 

North Carolina 
M SD 

Oregon 
M SD 

Obtained t 

School Enrollment (Number) 3 

400.20 

70.93 

354.75 

154.41 

0.59 

Classroom Teachers (FTE) a 

27.08 

4.73 

16.99 

6.27 

2.76 

Student/Teacher Ratio 3 

14.84 

1.69 

21.02 

4.09 

-3.11 

School Enrollment 3 (Percent) 

Kindergarten 

15.61 

1.51 

15.63 

3.87 

-0.01 

First 

14.41 

2.41 

17.29 

3.91 

-1.32 

Second 

14.83 

2.68 

15.57 

0.54 

-0.45 

Third 

16.71 

2.51 

15.29 

1.41 

0.88 

Fourth 

16.26 

2.28 

21.31 

9.31 

-1.19 

Fifth 

16.41 

1.06 

18.76 

9.29 

-0.57 

School Free Lunch Eligible 3 

253.40 

65.63 

160.75 

87.54 

1.82 

School Reduced-Price Lunch Eligible 3 

31.40 

6.12 

21.00 

7.02 

2.38 

Student Ethnicity 3 (Percent) 

African American 

45.29 

27.12 

1.34 

0.71 

3.20 

American Indian 

0.84 

0.31 

1.11 

0.80 

-0.64 

Asian 

4.88 

4.66 

2.10 

1.53 

1.13 

Hispanic 

16.13 

6.99 

24.63 

12.92 

-1.27 

Caucasian 

28.28 

24.53 

64.94 

16.48 

-2.55 

Two or More Races 

5.01 

2.94 

5.14 

2.39 

-0.07 

Student Gender 3 (Percent) 

Male 

49.91 

2.41 

50.68 

3.08 

-0.43 

Number of PBIS Team Members 

11.80 

1.30 

8.25 

3.86 

1.95 


Note. No statistically significant differences ip < .01) were found between states. 
Retrieved from http://nces.ed.gov/ccd/schoolsearch/ 
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Table 2 

Cross-Tabulation of DORA-II Foundations and Recommended Core Features of Effective 
Meetings 


DORA-II Foundations Recommended Core Features 



Anderson 

Perkins 

Haynes 1 

Tobia 1 

Timm i 

L Start meeting on time... 

X 

X 

X 

X 

X 

2. Share previous minutes . . . 

X 

X 

X 

X 

X 

3. Have agenda available. .. 

X 

X 

X 

X 

X 

4. Have clearly-defined 
roles... 

N/A 

X 

X 

X 

X 

5. Share data to support 
discussion ... 

X 

X 

X 

X 

X 

6. Schedule follow-up 
meeting... 

X 

X 

X 

X 

X 

7. End meeting on time... 

X 

X 

X 

X 

X 

Percent of Recommended 
Core Features 

86 

100 

100 

100 

100 

^Haynes (1988); "Tobia & Becker, (1990); 

-Timm (1997) 
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Table 3 

Cross-Tabulation of DORA-II Problem-Solving Processes and Recommended Principles from 
Practice 


DORA-II Problem 
Solving Process 


Recommended Principles from Practice 



Gilbert 

Bransford 1 

Deno 

2 

Boudett 

Hamilton 

1. Identify problem with 
precision 

X 

X 

X 

X 

X 

2. Identify goal for change 

3. Identify solution and 

X 

X 

X 

X 

X 

create implementation 
plan with contextual 
fit 

X 

X 

X 

X 

X 

4. Implement solution 
with high fidelity 

X 

X 

X 

X 

X 

5. Monitor impact of 
solution and compare 
against goal 

X 

X 

X 

X 

X 

6. Make summative 
evaluation decision 

X 

X 

X 

X 

X 

Percent of Principles from 
Practice 

r in n, • / i aao\ 2t 

100 

V 1 /~1 • 

100 

. n /r /t 

100 

“inn/" i \ 

100 

3t T • 1 , X T 1 

100 


; Brans ford & Stein (1993); Bouden, City, & Murnane (2006a. b); ’Hamilton. Halverson, 


Jackson, Mandinach, Supovitz, & Wayman (2009). 



MEASURING TEAM PROBLEM SOLVING 


35 


Table 4 


Inter-Obser\’er Agreement across TIPS/DORA-11 Categories 


Category 

Average 

Minimum 

Maximum 

Foundations 

97.00% 

80.00% 

100.00% 

Problem Identification 

86.39% 

50.00% 

100.00% 

Problem Precision 

82.95% 

20.00% 

100.00% 

Quantitative Data Use 

100.00% 

100.00% 

100.00% 

Goal Identification 

96.71% 

50.00% 

100.00% 

Solution Implementation Integrity Plan 

88.00% 

60.00% 

100.00% 

Solution Implementation Status 

74.44% 

0.00% 

100.00% 

Status of Problem Reported 

89.44% 

0.00% 

100.00% 

Status of Problem Compared Against Goal 

97.78% 

66.67% 

100.00% 

Decision After Status of Problem Reported 

88.89% 

0.00% 

100.00% 

Average 90.16% 
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Figure 1 

Team-Initiated Problem Solving (TIPS) Model 
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Appendix A 

2013-07-18 - 

DORA II Page No. 


DORA (Decision Observation, Recording, and Analysis) II 



School ID No.: 

Section 1. Demographic Information 

# PBIS Team Members: Observer Name: 

^Primary Observer 

1 Reliability Observer 

Date: 

Scheduled Start Time: Scheduled End Time: 




If these are research data complete the following: 

Group: State: 

Condition: 

Data Wave No.: 


Section 2. Foundations of Effective Team Problem Solving 

START OF MEETING 

DURING MEETING (ROLES) 

END OF MEETING 

01. O Meeting started within 10 minutes of scheduled start time 

02. O At least 75% of team members present at the start of the meeting 

03. O Previous meeting minutes available 

04. O Agenda available 

05. | | Facilitator 

06. O Minute Taker 

07. O Data Analyst 

08. O Next meeting scheduled 

09. O Meeting ended within 10 minutes of scheduled end time 

(includes a revised end time that team members agreed to) 

10. O At least 75% of team members present at the end of the 

meeting 


Notes 
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Section 3. Team Problem-Solving Processes 

Operational definition of a “problem” - At least one team member or meeting participant Q identifies a student social or academic behavior to change, AND Q the team selects/selected a 
solution to bring about the desired change. 

PRECISE PROBLEM & GOAL FOR 
CHANGE IDENTIFIED 

SOLUTION IDENTIFIED AND IMPLEMENTATION 
PLAN CREATED & SOLUTION IMPLEMENTED 
WITH INTEGRITY 

IMPACT OF SOLUTION MONITORED AND COMPARED 
AGAINST GOAL & SUMMATIVE EVALUATION DECISION 

MADE 

Problem PR 

Description of selected solution 

5. Status of Problem Reported - Direction of Change 

No. 

Write description of Solution below, including its 
individual components or “solution actions.” 

& Relation of Change to Goal 


NA Imprv. Imprv. 

Description of identified problem 


(See No but not & Met Not 



Protocol) Worse Change to Goal Goal Unclear Reported 

□ □□□□□ □ 



Description of status of problem (i.e., summary of findings from 
qualitative and/or quantitative data) or NA if New Problem 




1.1 Problem Precision 



What Who Where When Why 

□ □ □ □ □ 



Problem Category 



Social Behavior Academic Behavior 



□ 

□ 

3. Solution Implementation Plan 



NA Integrity 


Problem Features 

Old Imp. Integrity When Integrity 

Data source for report on status of problem (NA if New Problem) 

New Old Individual Group 

□ □ □ □ 

Prob. Person Timeline What G/R Who 

□ □□□□□ 


Description of implementation integrity plan 


1.2 Quantitative Data Use 



Social Behavior Academic Behavior 



□ 

□ 


6. Summative Evaluation Decision 

Description of data presented 


(Note: Check “No” if team did not report status of problem) 



(Note: Examples of summative evaluation decisions include (a) 
retaining, revising, or terminating (a) the solution, (b) the goal, (c) the 
precisely-defined problem, or (d) some combination of the preceding) 

2. Identified Goal 

4. Solution Implementation-Integrity 

NA New Problem Yes No 

What Change By When 

NA Not 

□ 

□ 

□ 

□ □ 

New Not Part. Imp. w/ reported/ 

Description of decision or NA if New Problem 

Description of change to be achieved 

Prob. Started Imp. Integrity Stopped. DK 


1 1 Postponed/out of time 
QOld problem not discussed 

□ □ □ □ □ □ 
1 1 Postponed/out of time 

1 1 Postponed/out of time 






